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Abstract 

This paper presents a corpus-based approach to 
word sense disambiguation where a decision tree as- 
signs a sense to an ambiguous word based on the 
bigrams that occur nearby. This approach is evalu- 
ated using the sense-tagged corpora from the 1998 
SENSEVAL word sense disambiguation exercise. It 
is more accurate than the average results reported 
for 30 of 36 words, and is more accurate than the 
best results for 19 of 36 words. 

1 Introduction 

Word sense disambiguation is the process of selecting 
the most appropriate meaning for a word, based on 
the context in which it occurs. For our purposes it is 
assumed that the set of possible meanings, i.e., the 
sense inventory, has already been determined. For 
example, suppose bill has the following set of possi- 
ble meanings: a piece of currency, pending legisla- 
tion, or a bird jaw. When used in the context of The 
Senate bill is under consideration, a human reader 
immediately understands that bill is being used in 
the legislative sense. However, a computer program 
attempting to perform the same task faces a difficult 
problem since it does not have the benefit of innate 
common-sense or linguistic knowledge. 

Rather than attempting to provide computer pro- 
grams with real-world knowledge comparable to 
that of humans, natural language processing has 
turned to corpus-based methods. These approaches 
use techniques from statistics and machine learn- 
ing to induce models of language usage from large 
samples of text. These models are trained to per- 
form particular tasks, usually via supervised learn- 
ing. This paper describes an approach where a deci- 
sion tree is learned from some number of sentences 
where each instance of an ambiguous word has been 
manually annotated with a sense-tag that denotes 
the most appropriate sense for that context. 

Prior to learning, the sense-tagged corpus must be 
converted into a more regular form suitable for auto- 
matic processing. Each sense-tagged occurrence of 
an ambiguous word is converted into a feature vec- 
tor, where each feature represents some property of 



the surrounding text that is considered to be relevant 
to the disambiguation process. Given the flexibility 
and complexity of human language, there is poten- 
tially an infinite set of features that could be utilized. 
However, in corpus-based approaches features usu- 
ally consist of information that can be readily iden- 
tified in the text, without relying on extensive exter- 
nal knowledge sources. These typically include the 
part-of-speech of surrounding words, the presence 
of certain key words within some window of context, 
and various syntactic properties of the sentence and 
the ambiguous word. 

The approach in this paper relies upon a feature 
set made up of bigrams, two word sequences that 
occur in a text. The context in which an ambiguous 
word occurs is represented by some number of binary 
features that indicate whether or not a particular 
bigram has occurred within approximately 50 words 
to the left or right of the word being disambiguated. 

We take this approach since surface lexical fea- 
tures like bigrams, collocations, and co-occurrences 
often contribute a great deal to disambiguation ac- 
curacy. It is not clear how much disambiguation ac- 
curacy is improved through the use of features that 
are identified by more complex pre-processing such 
as part-of-speech tagging, parsing, or anaphora res- 
olution. One of our objectives is to establish a clear 
upper bounds on the accuracy of disambiguation us- 
ing feature sets that do not impose substantial pre- 
processing requirements. 

This paper continues with a discussion of our 
methods for identifying the bigrams that should be 
included in the feature set for learning. Then the 
decision tree learning algorithm is described, as are 
some benchmark learning algorithms that are in- 
cluded for purposes of comparison. The experimen- 
tal data is discussed, and then the empirical results 
are presented. We close with an analysis of our find- 
ings and a discussion of related work. 

2 Building a Feature Set of Bigrams 

We have developed an approach to word sense dis- 
ambiguation that represents text entirely in terms of 
the occurrence of bigrams, which we define to be two 



data distributions. However, (Cressie and Read 
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Figure 1: Representation of Bigram Counts 



consecutive words that occur in a text. The distri- 
butional characteristics of bigrams are fairly consis- 
tent across corpora; a majority of them only occur 
one time. Given the sparse and skewed nature of 
this data, the statistical methods used to select in- 
teresting bigrams must be carefully chosen. We ex- 
plore two alternatives, the power divergence family 
of goodness of fit statistics and the Dice Coefficient, 
an information theoretic measure related to point- 
wise Mutual Information. 

Figure ^ summarizes the notation for word and 
bigram counts used in this paper by way of a 2 x 2 
contingency table. The value of rin shows how many 
times the bigram big cat occurs in the corpus. The 
value of 77,12 shows how often bigrams occur where 
big is the first word and cat is not the second. The 
counts in 77+1 and 711+ indicate how often words big 
and cat occur as the first and second words of any 
bigram in the corpus. The total number of bigrams 
in the corpus is represented by 77,++. 

2.1 The Power Divergence Family 



(Cressie and Read, 1984) introduce the power diver- 
gence family of goodness of fit statistics. A number 
of well known statistics belong to this family, includ- 
ing the likelihood ratio statistic and Pearson's 
statistic. 

These measure the divergence of the observed 
(riij) and expected {rriij) bigram counts, where rriij 
is estimated based on the assumption that the com- 
ponent words in the bigram occur together strictly 
by chance: 



rely on the distributional assumptions that underly 
both Pearson's test and the likelihood ratio. 

Unfortunately it is usually not clear which test 
is most appropriate for a particular sample of data. 
We take the following approach, based on the obser- 
vation that all tests should assign approximately the 
same measure of statistical significance when the bi- 
gram counts in the contingency table do not violate 
any of the distributional assumptions that underly 
the goodness of fit statistics. We perform tests us- 
ing X^, G^, and Fisher's exact test for each bigram. 
If the resulting measures of statistical significance 
differ, then the distribution of the bigram counts is 
causing at least one of the tests to become unreli- 
able. When this occurs we rely upon the value from 
Fisher's exact test since it makes fewer assumptions 
about the underlying distribution of data. 

For the experiments in this paper, we identified 
the top 100 ranked bigrams that occur more than 5 
times in the training corpus associated with a word. 
There were no cases where rankings produced by 
G^, X^, and Fisher's exact test disagreed, which is 
not altogether surprising given that low frequency 
bigrams were excluded. Since all of these statistics 
produced the same rankings, hereafter we make no 
distinction among them and simply refer to them 
generically as the power divergence statistic. 

2.2 Dice Coefficient 

The Dice Coefficient is a descriptive statistic that 
provides a measure of association among two words 
in a corpus. It is similar to pointwise Mutual Infor- 
mation, a widely used measure that was fir st intro- 
duced for identifyi ng lexical relationships in (Church 
and Hanks, 1990). Pointwise Mutual Information 



71,j+ * 77+j 



can be defined as follows: 



MI{wi,W2) — log2 



nil * n++ 
77+1 *ni+ 



Given this value, G^ and X"^ are calculated as: 



^2^77y ^log- 
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(Dunning, 1993) argues in favor of G^ over X^, es- 
pecially when dealing with very sparse and skewed 



where wi and W2 represent the two words that make 
up the bigram. 

Pointwise Mutual Information quantifies how of- 
ten two words occur together in a bigram (the nu- 
merator) relative to how often they occur overall in 
the corpus (the denominator). However, there is a 
curious limitation to pointwise Mutual Information. 
A bigram W1W2 that occurs Tin times in the corpus, 
and whose component words wi and ■ii;2 only occur 
as a part of that bigram, will result in increasingly 
strong measures of association as the value of 77,11 
decreases. Thus, the maximum pointwise Mutual 



Information in a given corpus will be assigned to bi- 
grams that occur one time, and whose component 
words never occur outside that bigram. These are 
usually not the bigrams that prove most useful for 
disambiguation, yet they will dominate a ranked list 
as determined by pointwise Mutual Information. 

The Dice Coefficient overcomes this limitation, 
and can be defined as follows: 



Dice(wi,W2) 



2 * Till 



When nil = ni+ = n+i the value of Dice{wi, W2) 
will be 1 for all values nu. When the value of nn 
is less than either of the marginal totals (the more 
typical case) the rankings produced by the Dice Co- 
efficient are similar to those of Mutual Information. 
The relationship between pointwise Mutual Infor- 
mation and the Dice Coefficient is also discussed in 



( Smadja et al., 1996 ). 

We have developed the Bigram Statistics Package 
to produce ranked lists of bigrams using a range of 
tests. This software is written in Perl and is freely 
available from www. d. umn.edu/~tpederse. 

3 Learning Decision Trees 

Decision trees are among the most widely used ma- 
chine learning algorithms. They perform a general 
to specific search of a feature space, adding the most 
informative features to a tree structure as the search 
proceeds. The objective is to select a minimal set of 
features that efficiently partitions the feature space 
into classes of observations and assemble them into 
a tree. In our case, the observations are manually 
sense-tagged examples of an ambiguous word in con- 
text and the partitions correspond to the different 
possible senses. 

Each feature selected during the search process is 
represented by a node in the learned decision tree. 
Each node represents a choice point between a num- 
ber of different possible values for a feature. Learn- 
ing continues until all the training examples are ac- 
counted for by the decision tree. In general, such 
a tree will be overly specific to the training data 
and not generalize well to new examples. Therefore 
learning is followed by a pruning step where some 
nodes are eliminated or reorganized to produce a 
tree that can generalize to new circumstances. 

Test instances are disambiguated by finding a path 
through the learned decision tree from the root to a 
leaf node that corresponds with the observed fea- 
tures. An instance of an ambiguous word is dis- 
ambiguated by passing it through a series of tests, 
where each test asks if a particular bigram occurs in 
the available window of context. 

We also include three benchmark learning algo- 
rithms in this study: the majority classifier, the de- 



cision stump, and the Naive Bayesian classifier. 

The majority classifier assigns the most common 
sense in the training data to every instance in the 
test data. A dec ision stump is a one node decision 
trec( Holte, 1993 ) that is created by stopping the de- 
cision tree learner after the single most informative 
feature is added to the tree. 



Th e Naive Bayesian classifier ( Duda and Hart ^ 
1973| ) is based on certain blanket assumptions about 



the interactions among features in a corpus. There 
is no search of the feature space performed to build 
a representative model as is the case with decision 
trees. Instead, all features are included in the classi- 
fier and assumed to be relevant to the task at hand. 
There is a further assumption that each feature is 
conditionally independent of all other features, given 
the sense of the ambiguous word. It is most often 
used with a bag of words feature set, where every 
word in the training sample is represented by a bi- 
nary feature that indicates whether or not it occurs 
in the window of context surrounding the ambiguous 
word. 

We use the Weka ( Witten and Frank, 2000| ) imple- 
mentations of the C4.5 decision tree learner (known 
as J48), the decision stump, and the Naive Bayesian 
classifier. Weka is written in Java and is freely avail- 
able from www.cs.waikato.ac.nz/~ml. 

4 Experimental Data 

Our empirical study utilizes the training and test 
data from the 1998 SENSE VAL evaluation of word 
sense disambiguation systems. Ten teams partic- 
ipated in the supervised learning portion of this 
event. Additional details about the exercise, in- 
cluding the data and results referred to in this 
paper, can be found at the SENSEVAL web site 



(www . it ri . bton . ac . uk/eve nt s /senseval / ) and in (Kil 
^arriff and Palmer, 200C ) 



We included all 36 tasks from SENSEVAL for 
which training and test data were provided. Each 
task requires that the occurrences of a particular 
word in the test data be disambiguated based on 
a model learned from the sense-tagged instances in 
the training data. Some words were used in multiple 
tasks as different parts of speech. For example, there 
were two tasks associated with bet, one for its use as 
a noun and the other as a verb. Thus, there are 
36 tasks involving the disambiguation of 29 different 
words. 

The words and part of speech associated with each 
task are shown in Table |l| in column 1. Note that 
the parts of speech are encoded as n for noun, a 
for adjective, v for verb, and p for words where the 
part of speech was not provided. The number of 
test and training instances for each task are shown 
in columns 2 and 4. Each instance consists of the 
sentence in which the ambiguous word occurs as well 



as one or two surrounding sentences. In general the 
total context available for each ambiguous word is 
less than 100 surrounding words. The number of 
distinct senses in the test data for each task is shown 
in column 3. 

5 Experimental Method 

The following process is repeated for each task. Cap- 
italization and punctuation are removed from the 
training and test data. Two feature sets are selected 
from the training data based on the top 100 ranked 
bigrams according to the power divergence statistic 
and the Dice Coefficient. The bigram must have oc- 
curred 5 or more times to be included as a feature. 
This step filters out a large number of possible bi- 
grams and allows the decision tree learner to focus 
on a small number of candidate bigrams that are 
likely to be helpful in the disambiguation process. 

The training and test data are converted to fea- 
ture vectors where each feature represents the occur- 
rence of one of the bigrams that belong in the feature 
set. This representation of the training data is the 
actual input to the learning algorithms. Decision 
tree and decision stump learning is performed twice, 
once using the feature set determined by the power 
divergence statistic and again using the feature set 
identified by the Dice Coefficient. The majority clas- 
sifier simply determines the most frequent sense in 
the training data and assigns that to all instances 
in the test data. The Naive Bayesian classifier is 
based on a feature set where every word that occurs 

5 or more times in the training data is included as a 
feature. 

All of these learned models are used to disam- 
biguate the test data. The test data is kept separate 
until this stage. We employ a fine grained scoring 
method, where a word is counted as correctly disam- 
biguated only when the assigned sense tag exactly 
matches the true sense tag. No partial credit is as- 
signed for near misses. 

6 Experimental Results 

The accuracy attained by each of the learning algo- 
rithms is shown in Table [T]. Column 5 reports the 
accuracy of the majority classifier, columns 6 and 7 
show the best and average accuracy reported by the 
10 participating SENSEVAL teams. The evaluation 
at SENSEVAL was based on precision and recall, so 
we converted those scores to accuracy by taking their 
product. However, the best precision and recall may 
have come from different teams, so the best accuracy 
shown in column 6 may actually be higher than that 
of any single participating SENSEVAL system. The 
average accuracy in column 7 is the product of the 
average precision and recall reported for the par- 
ticipating SENSEVAL teams. Column 8 shows the 
accuracy of the decision tree using the J48 learning 



algorithm and the features identified by a power di- 
vergence statistic. Column 10 shows the accuracy 
of the decision tree when the Dice Coefficient selects 
the features. Columns 9 and 11 show the accuracy 
of the decision stump based on the power divergence 
statistic and the Dice Coefficient respectively. Fi- 
nally, column 13 shows the accuracy of the Naive 
Bayesian classifier based on a bag of words feature 
set. 

The most accurate method is the decision tree 
based on a feature set determined by the power di- 
vergence statistic. The last line of Table |l| shows 
the win-tie-loss score of the decision tree/power di- 
vergence method relative to every other method. A 
win shows it was more accurate than the method in 
the column, a loss means it was less accurate, and 
a tie means it was equally accurate. The decision 
tree/power divergence method was more accurate 
than the best reported SENSEVAL resuhs for 19 
of the 36 tasks, and more accurate for 30 of the 36 
tasks when compared to the average reported accu- 
racy. The decision stumps also fared well, proving to 
be more accurate than the best SENSEVAL results 
for 14 of the 36 tasks. 

In general the feature sets selected by the power 
divergence statistic result in more accurate decision 
trees than those selected by the Dice Coefficient. 
The power divergence tests prove to be more reliable 
since they account for all possible events surround- 
ing two words wi and W2; when they occur as bigram 
z/;iU'2, when wi or W2 occurs in a bigram without the 
other, and when a bigram consists of neither. The 
Dice Coefficient is based strictly on the event where 
wi and W2 occur together in a bigram. 

There are 6 tasks where the decision tree / power 
divergence approach is less accurate than the SEN- 
SEVAL average; promise-n, scrap-n, shirt-n, amaze- 
V, bitter-p, and sanction-p. The most dramatic dif- 
ference occurred with amaze-v, where the SENSE- 
VAL average was 92.4% and the decision tree accu- 
racy was 58.6%. However, this was an unusual task 
where every instance in the test data belonged to a 
single sense that was a minority sense in the training 
data. 

7 Analysis of Experimental Results 

The characteristics of the decision trees and deci- 
sion stumps learned for each word are shown in 
Table |^. Column 1 shows the word and part of 
speech. Columns 2,3, and 4 are based on the feature 
set selected by the power divergence statistic while 
columns 5, 6, and 7 are based on the Dice Coeffi- 
cient. Columns 2 and 5 show the node selected to 
serve as the decision stump. Columns 3 and 6 show 
the number of leaf nodes in the learned decision tree 
relative to the number of total nodes. Columns 4 
and 7 show the number of bigram features selected 



Table 1: Experimental Results 
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to represent the training data. 

This table shows that there is little difference in 

the decision stump nodes selected from feature sets 
determined by the power divergence statistics versus 
the Dice Coefficient. This is to be expected since the 
top ranked bigrams for each measure are consistent, 
and the decision stump node is generally chosen from 
among those. 

However, there are differences between the feature 
sets selected by the power divergence statistics and 



the Dice Coefficient. These are reflected in the dif- 
ferent sized trees that are learned based on these 
feature sets. The number of leaf nodes and the total 
number of nodes for each learned tree is shown in 
columns 3 and 6. The number of internal nodes is 
simply the difference between the total nodes and 
the leaf nodes. Each leaf node represents the end 
of a path through the decision tree that makes a 
sense distinction. Since a bigram feature can only 
appear once in the decision tree, the number of inter- 



Table 2: Decision Tree and Stump Characteristics 
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20/39 


106 


bury-v 


buried in 


28/55 


103 


buried in 


32/63 


103 


calculate-v 


calculated to 


5/9 


103 


calculated to 


5/9 


103 


consume-v 


on the 


4/7 


20 


on the 


4/7 


20 


derive-v 


derived from 


10/19 


104 


derived from 


10/19 


104 


fioat-v 


floated on 


24/47 


80 


floated on 


24/47 


80 


invade-v 


to invade 


55/109 


107 


to invade 


66/127 


108 


promise-v 


promise to 


3/5 


100 


promise you 


5/9 


106 


sack-v 


return to 


1/1 


91 


return to 


1/1 


91 


scrap-v 


of the 


1/1 


7 


of the 


1/1 


7 


seize-v 


to seize 


26/51 


104 


to seize 


57/113 


104 


brilhant-a 


a brilliant 


26/51 


101 


a brilliant 


42/83 


103 


floating-a 


in the 


7/13 


10 


in the 


7/13 


10 


generous- a 


a generous 


57/113 


103 


a generous 


56/111 


102 


giant-a 


the giant 


2/3 


102 


a giant 


1/1 


101 


modest-a 


a modest 


14/27 


101 


a modest 


10/19 


105 


shght-a 


the slightest 


2/3 


105 


the slightest 


2/3 


105 


wooden-a 


wooden spoon 


2/3 


104 


wooden spoon 


2/3 


101 


band-p 


band of 


14/27 


100 


the band 


21/41 


117 


bitter-p 


a bitter 


22/43 


54 


a bitter 


22/43 


54 


sanction-p 


south africa 


12/23 


52 


south africa 


12/23 


52 


shake- p 


his head 


90/179 


100 


his head 


81/161 


105 



nal nodes represents the number of bigram features 
selected by the decision tree learner. 

One of our original hypotheses was that accurate 
decision trees of bigrams will include a relatively 
small number of features. This was motivated by 
the success of decision stumps in performing disam- 
biguation based on a single bigram feature. In these 
experiments, there were no decision trees that used 
all of the bigram features identified by the filtering 
step, and for many words the decision tree learner 
went on to eliminate most of the candidate features. 
This can be seen by comparing the number of inter- 
nal nodes with the number of candidate features as 



shown in columns 4 or 7.|j 

It is also noteworthy that the bigrams ultimately 
selected by the decision tree learner for inclusion in 
the tree do not always include those bigrams ranked 
most highly by the power divergence statistic or the 
Dice Coefficient. This is to be expected, since the 
selection of the bigrams from raw text is only mea- 



For most words the 100 top ranked bigrams form the set 
of candidate features presented to the decision tree learner. If 
there are ties in the top 100 rankings then there may be more 
than 100 features, and if the there were fewer than 100 bi- 
grams that occurred more than 5 times then all such bigrams 
are included in the feature set. 



suring the association between two words, while the 



decision tree seeks bigrams that partition instances 



of the ambiguous word into into distinct senses. In 
particular, the decision tree learner makes decisions 
as to what bigram to include as nodes in the tree 
using the gain ratio, a measure based on the over- 
all Mutual Information between the bigram and a 
particular word sense. 

Finally, note that the smallest decision trees are 
functionally equivalent to our benchmark methods. 
A decision tree with 1 leaf node and no internal 
nodes (1/1) acts as a majority classifier. A deci- 
sion tree with 2 leaf nodes and 1 internal node (2/3) 
has the structure of a decision stump. 

8 Discussion 

One of our long-term objectives is to identify a core 
set of features that will be useful for disambiguat- 
ing a wide class of words using both supervised and 
unsupervised methodologies. 



of the bigram (e. g., (Bruce and Wiebe, 19941 ), (Ng 
and Lee, 199^ ), ( |Yarowsky, 1995]) ). WhIIe~some of 



We have presented an ensemble approach to word 



sense disambiguation ( Pedersen, 2000| ) where mul- 
tiple Naive Bayesian classifiers, each based on co- 
occurrence features from varying sized windows of 
context, is shown to perform well on the widely stud- 
ied nouns interest and line. While the accuracy of 
this approach was as good as any previously pub- 
lished results, the learned models were complex and 
difficult to interpret, in effect acting as very accurate 
black boxes. 

Our experience has been that variations in learn- 
ing algorithms are far less significant contributors 
to disambiguation accuracy than are variations in 
the feature set. In other words, an informative fea- 
ture set will result in accurate disambiguation when 
used with a wide range of learning algorithms, but 
there is no learning algorithm that can perform well 
given an uninformative or misleading set of features. 
Therefore, our focus is on developing and discover- 
ing feature sets that make distinctions among word 
senses. Our learning algorithms must not only pro- 
duce accurate models, but they should also shed new 
light on the relationships among features and allow 
us to continue refining and understanding our fea- 
ture sets. 

We believe that decision trees meet these criteria. 
A wide range of implementations are available, and 
they are known to be robust and accurate across a 
range of domains. Most important, their structure 
is easy to interpret and may provide insights into 
the relationships that exist among features and more 
general rules of disambiguation. 

9 Related Work 

Bigrams have been used as features for word sense 
disambiguation, particularly in the form of colloca- 
tions where the ambiguous word is one component 



the bigrams we identify are collocations that include 
the word being disambiguated, there is no require- 
ment that this be the case. 

Decision trees have been used in supervised learn- 
ing approaches to word sense disambiguation, and 
have f ared well in a nu mber of comparative studi es 
(e.g., ( [Mooney, 1996| ), ( [Pedersen and Bruce, 1997| )). 
In the former they were used with the bag of word 
feature sets and in the latter they were used with a 
mixed feature set that included the part-of-speech of 
neighboring words, three collocations, and the mor- 
phology of the ambiguous word. We believe that 
the approach in this paper is the first time that de- 
cision trees based strictly on bigram features have 
been employed. 

The decision list is a closely related approach that 
has also b een applied to wo rd sense disambigua- 



tion (e. g., (Yarowsky, 1994 ), ( Wilks and Stevenson , 
1998|) , jYarowsky, 2000| )). Rather than building and 



traversing a tree to perform disambiguation, a list is 
employed. In the general case a decision list may suf- 
fer from less fragmentation during learning than de- 
cision trees; as a practical matter this means that the 
decision list is less likely to be over-trained. How- 
ever, we believe that fragmentation also reflects on 
the feature set used for learning. Ours consists of 
at most approximately 100 binary features. This re- 
sults in a relatively small feature space that is not 
as likely to suffer from fragmentation as are larger 
spaces. 

10 Future Work 

There are a number of immediate extensions to this 
work. The first is to ease the requirement that bi- 
grams be made up of two consecutive words. Rather, 
we will search for bigrams where the component 
words may be separated by other words in the text. 
The second is to eliminate the filtering step by which 
candidate bigrams are selected by a power diver- 
gence statistic. Instead, the decision tree learner 
would consider all possible bigrams. Despite increas- 
ing the danger of fragmentation, this is an interest- 
ing issue since the bigrams judged most informative 
by the decision tree learner are not always ranked 
highly in the filtering step. In particular, we will 
determine if the filtering process ever eliminates bi- 
grams that could be significant sources of disam- 
biguation information. 

In the longer term, we hope to adapt this approach 
to unsupervised learning, where disambiguation is 
performed without the benefit of sense tagged text. 
We are optimistic that this is viable, since bigram 
features are easy to identify in raw text. 



11 Conclusion 

This paper shows that the combination of a simple 
feature set made up of bigrams and a standard deci- 
sion tree learning algorithm results in accurate word 
sense disambiguation. The results of this approach 
are compared with those from the 1998 SENSEVAL 
word sense disambiguation exercise and show that 
the bigram based decision tree approach is more ac- 
curate than the best SENSEVAL results for 19 of 36 
words. 
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